
II " i rl"-" 1 "" Tr adfmark Office 



P.O.Box 1450 , . ,,< n 
Alexandria, Virginia 2231 3-1450 
www.uspto.gov 



7590 02/09/2004 

DONALD L. BARTELS 
COUDERT BROTHERS LLP 
600 BEACH STREET, 3rd FLOOR 
SAN FRANCISCO, CA 94109 



2175 

DATE MAE-ED: 02/09/2004 




PTO-90C (Rev. 10/03) 



Office Action Summary 



Application No. 

09/884,504 
Examiner 



Applicant(s) 

LAYEGHI, SHAHAB 
Art Unit 
2175 



Apu M Mofiz _ i . -•— 

'^SHORTENED SWUTORV PERIOO FOR REPLY ,S SET TO EXPIRE 3 MONTH(S) PROM 
THE MAILING DATE OF THIS COMMUNICATION.^^ , n nQ event> howeve , may a reply be timely filed 

I ermed P patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 M Responsive to communication(s) filed on UJuneJOOl- 

. 2b^ This action is non-final. 

2a)D This action is FINAL. i** ma « A ™ nrosecution as to the merits is 

Disposition of Claims 

4^ Claim(s) 1-23 is/are pending in the application. 

m 4a) Of thelTove claim(s) is/are withdrawn from cons.deration. 

5) D Claim(s) is/are allowed. 

6) E3 .Claim(s) 1=23 is/are rejected. 
7 \r-| ciaim(s) is/are objected to. 

Application Papers 

<,)□ The specification is objected to by the E«aminer_ ^ ^ Exam|ner 

,0)K1 The drawing(s) filed on UMmJQBl is/are. a® accepted »b L J 

Applicant may no. revest that any objeCon to «h, drawing,*) „. ^ 37 CFR 1.121*1). 
R e pl3cemen,dra* g sbe^ 
1 1 )Q The oath or declaration is obiected to by the Examiner, mo 
Priority under 35 U.S.C.§§ 119 and 120 n 9 (a)-(d) or (f). 

,2)Q Acknowledgment is made of a claim for foreign pnonty under 35 U.S.C. § 1* 

a »° S Sied^Sth^y document, have been received, 

2 3 : B sr-TKSs sss ssgnr^ in mis Na,ional s,a9e 

.SeetheSSrS 
S translation oftheforeign language ^^^^^L ,21 since a specific 



Attachment(s) 

1)S Notice of References Cited (PTO-892) 
2 □ Notice of Drattsperson's Patent Draw.ng Rev.ew (PTO-948 
3) y formation Disdosure Statement(s) (PTO-1449) Paper No(s) 4&6 . 



4) □ Interview Summary (PTO-413) Paper No(s)^ • 

5) □ Notice of informal Patent Application (PTO-152) 

6) O Other: 



U S Patent and Trademark Office 

PTOL-326 (Rev. 11-03) 



Office Action Summary 



Part of Paper No. 7 



Application/Control Number: 09/884,504 Page 2 
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DETAILED ACTION 
Claim Rejections - 35 USC § 103 

1 . The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

2. Claims 1-13, 22 and 23 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Purnaveja et al. (U.S. Patent No. 6,006,241 and Purnaveja 
hereinafter) in view of Tao (U.S. Patent No. 6,243,481 ). 

As to claim 1 , Purnaveja teaches an encoder (col 6, lines 25-35) for encoding a 

media file (i.e. "a previously digitized video file 314, e.g., a Video for Windows (.avijfile. ") (col 2, lines 25-67) 
SO that it is Capable Of being decoded (i.e. "Next, the encoded video/audio streams are decoded by the 
decoder 964, i.e., decompressed using a suitable technique, and then displayed at client computer 240 by renderer 
965. ") (col 9, lines 1-10) for playback as a media Stream (i.e. "The video/audio streams and annotation 
streams are stored in stream server(s) for subsequent retrieval by client computers) in a coordinated manner, so 
that the client computers) is able to synchronously display the video frames and displayable event(s) in a video 
window and event windows), respectively. ") (col 3, lines 1-20), a method for encoding data (i.e. 
annotation data) (Abstract) and time restriction information (i.e. "Another visual control aid, zoom bar 
716, enables designer 219 to select the respective portions of the complete time tracks 750, 760, 770, 780, and 790, 
and defined by start time indicator 712 and end time indicator 718, which is currently displayed by author tool 

700. ") ( col 7, lines 10-67) with said media file such that when said encoded media file is 

decoded for playback (i.e. when the video data/frames are displayed or audio data played (i.e. multimedia 
data) at the client computer) (Abstract; Fig.3; Fig.4B; Fig.SB; Fig.9; Fig.lOA) 3S Said media Stream, said 
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data is reproduced in said media stream only during a time period predetermined (i.e. 

"For example, URL addresses (event locators) pointing to HTML pages enable client computer 240 to subsequently 
retrieve textual and/or graphical elements to be displayed at predetermined time as defined by the time markers of 

the flipper stream. ") (col 7, lines 10-67) by said time restriction information, comprising the steps 
of: (1 ) receiving a media file (col 2, lines 25-67); (2) receiving at least one data file (i.e. "When 

author module 318 has completed building an annotation stream, e.g., the flipper stream, the annotation stream is 
given a file name and loaded into a convenient server, e.g., stream server 220, for subsequent retrieval by client 
computer 240. ") (col 7, lines 10-67) Comprising at least One data Segment (i.e. the annotation frame) 

(Abstract; Fig.3; Fi g .4B; Fi g .8B; Fig.9; Fig.iOA; col 3, lines 1-25); (3) receiving time restriction 

information (col 7, lines 10-67) for each Said data Segment (Abstract; Fig.3; Fig.4B; Fig.SB; Fig.9; 

Fig.iOA; col 3, lines 1-25) that specifies said time period (col i, lines 10-67) when said data 

Segment (col 3, lines 1-25) is tO be reproduced as part Of (i.e. "In view of the foregoing, there are 
desired improved techniques for reliably providing a multimedia stream such as a video and audio stream, together 
with annotations such as textual and graphical information in an integrated seamless package to client computer (s), 
while efficiently utilizing the network resources and consuming minimal computational cycles on the client 

computer(s). ") (col 2, lines 25-67) said decoded media stream (col 3, lines 1-20); and (4) receiving 
encoded media file (col 2, lines 25-67); and said data segment (i.e. the annotation frames) (col 3, 
lines 1-25) and said corresponding time restriction information (col i, lines 10-67) such that 
said data segment (i.e. the annotation frame) (col 3, lines 1-25) is reproduced by a decoder (i.e. the 

decoder 964) (col 9, lines 1-10) during Said specified time period (i.e. "For example, URL addresses 
(event locators) pointing to HTML pages enable client computer 240 to subsequently retrieve textual and/or 
graphical elements to be displayed at predetermined time as defined by the time markers of the flipper stream. ") 

(col 7, lines 10-67). 
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Purnaveja does not explicitly teach embedding data segment as part of said 

encoded media data. 

Tao teaches embedding data segment as part of said encoded media data (i.e. 

"The present invention pertains to a method and system thereof for invisibly encoding and embedding information 
within digital data representing, for example, audio data, image data, and video data. " ... "In one embodime3nt, 
the digital data are in a compressed format (e.g. JPEG MPEG for image data, MP3 orAC3for audio data, as well 
as other data compression formats.) ") (Abstract; col 2, lines 28-45; col 3, lines 1-10). 

It would have been obvious to a person of ordinary skill in the art at the time of 
Applicant's invention to modify the teachings of Purnaveja with the teachings of Tao to 
include embedding data segment as part of said encoded media data with the 
motivation to embed multiple speech streams into a video, each speech stream in a 
different language, so that the video can be distributed to a wide range of users who 
can then each view the video in the language of their choosing (Tao, col 1, lines 30-35) and 
to store a large amount of information while being robust against noise (Tao, col 4, lines 33- 

36). 

As to claim 2, Purnaveja teaches that the time restriction information comprises a 
start time tag that specifies when said time period is to begin (i.e. "Another visual control aid, 

zoom bar 716, enables designer 219 to select the respective portions of the complete time tracks 750, 760, 770, 780, 
and 790, and defined by start time indicator 712 and end time indicator 718, which is currently displayed by author 

tool 700. ") (col 7, lines 10-67). 
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As to claim 3, Pgrnaveja teaches that the time restriction information comprises a 
stop time tag that specifies when said time period is to end (i.e. "Another visual control aid, 

zoom bar 716, enables designer 219 to select the respective portions of the complete time tracks 750, 760, 770, 780, 
and 790, and defined by start time indicator 712 and end time indicator 718, which is currently displayed by author 
tool 700. ") (col 7, lines 10-67). 

As to claim 4, Purnaveja teaches that the time restriction information comprises a 
start time tag that specifies when said time period is to begin and a stop time tag that 

Specifies When Said time period is tO end (i.e. "Another visual control aid, zoom bar 716, enables 
designer 219 to select the respective portions of the complete time tracks 750, 760, 770, 780, and 790, and defined * 
by start time indicator 712 and end time indicator 718, which is currently displayed by author tool 700. ") (col 7, 

lines 10-67). 

As to claim 5, Purnaveja teaches that the data is reproduced at a specified point 

in Said media Stream (i.e. "For example, URL addresses (event locators) pointing to HTML pages enable 
client computer 240 to subsequently retrieve textual and/or graphical elements to be displayed at predetermined 
time as defined by the time markers of the flipper stream. ") (col 7, lines 10-67). 

Purnaveja does not explicitly teach embedding data segment as part of said 

encoded media data. 

Tao teaches embedding data segment as part of said encoded media data (i.e. 

"The present invention pertains to a method and system thereof for invisibly encoding and embedding information 
within digital data representing, for example, audio data, image data, and video data. " ... "In one embodiment, 
the digital data are in a compressed format (e.g. JPEG MPEG for image data, MPS or ACS for audio data; as well 
as other data compression formats.)-) (Abstract; col 2, lines 28-45; col 3, lines 1-10). 
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It would have been obvious to a person of ordinary skill in the art at the time of 
Applicant's invention to modify the teachings of Purnaveja with the teachings of Tao to 
include embedding data segment as part of said encoded media data with the 
motivation to embed multiple speech streams into a video, each speech stream in a 
different language, so that the video can be distributed to a wide range of users who 
can then each view the video in the language of their choosing (Tao, col i, lines 30-35) and 
to store a large amount of information while being robust against noise (Tao, col 4, lines 33- 

36). 

As to claim 6, Purnaveja teaches that the data segment (col 3, lines 1-25) comprises 

a document file (i.e. "The first type of annotation streams are data annotation streams in which the displayable 
event data are embedded within the annotation streams. *' ... 'The second type of annotation streams are locator 
annotation streams in which the displayable data is either too cumbersome and/or is continually evolving to be 
embedded as static data within the annotation stream. Instead, event locator® pointing to the location of the 
displayable data are stored in the annotation streams instead of the displayable data. Examples include URL 
addresses pointing to HTML pages. ") (Examiner asserts that whether the data is embedded in the annotation stream 
or the event locator is pointing to the HTML pages, they are both document files) (col 6, lines 50-67). 

As to claim 7, Purnaveja teaches that the data segment (col 3, lines 1-25) comprises 

an Internet address (i.e. "The second type of annotation streams are locator annotation streams in which the 
displayable data is either too cumbersome and/or is continually evolving to be embedded as static data within the 
annotation stream. Instead, event locator(s) pointing to the location of the displayable data are stored in the 
annotation streams instead of the displayable data. Examples include URL addresses pointing to HTML pages. ") 
(col 6, lines 50-67). 
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As to claim 8, Purnaveja teaches that the media file is an audio file (i.e. -Annotated 

multimedia streams can include a compressed video stream for display in a video window, an accompanying 
compressed audio stream and annotations. "... "Sources for generating video streams include a video camera 312, 
a video cassette recorder (VCR) (not shown) or a previously digitized video file 314, e.g., a Video for Windows 
(.avi) file. ") (Examiner asserts that Purnaveja explicitly states that the system and the associated method are 
applicable to video, audio and image data. A video file example is given, but it is also applicable to audio media 

fiie(s)) (col 2, lines 35-67; col 5, lines 54-64) and said data segment (col 3, lines 1-25) is a non-audio 

data Segment (i.e. "The first type of annotation streams are data annotation streams in which the displayable 
event data are embedded within the annotation streams. Examples of data annotation streams include ticker tape 
data embedded within the annotation stream. ") (col 6, lines 25-67). 

AS tO Claim 9, Purnaveja teaches a audio file (i.e. "Annotated multimedia streams can 
include a compressed video stream for display in a video window, an accompanying compressed audio stream and 
annotations. "... "Sources for generating video streams include a video camera 312, a video cassette recorder 
(VCR) (not shown) or a previously digitized video file 314, e.g., a Video for Windows (avi) file. ") (Examiner asserts 
that Purnaveja explicitly states that the system and the associated method are applicable to video, audio and image 
data. A video file example is given, but it is also applicable to audio media file(s)) (col 2, lines 35-67; col 5, lines 

54-64). 

Purnaveja does not explicitly teach that the audio file is an MP3 file. 

TaO teaches that the audio file is an MP3 file (i.e. "the digital data are in a compressed 
format (e.g., JPEG or MPEG for image data, MP3 orAC3for audio data, as well as other data compression 
formats). ") (Abstract; col 3, lines 1-10). 
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It would have been obvious to a person of ordinary skill in the art at the time of 
Applicant's invention to modify the teachings of Purnaveja with the teachings of Tao to 
include that the audio file is an MP3 file with the motivation to provide for data that is 
robust against noise, easy to encode, detect, retrieve, decode and permits for relatively 
large amounts of hidden data to be encoded (Tao, col 12, lines 5-16). 

As to claim 1 0, Purnaveja teaches an encoder (Fig.3; Fi g .4B ; Fig.SB; Fig.9; Fig.lOA; col 6, 

lines 25-35) for encoding a media file (i.e. "a previously digitized video file 314, e.g., a Video for Windows 
(.avOfile. ") (col 2, lines 25-67) SO that it is Capable of being decoded (i.e. "Next, the encoded 
video/audio streams are decoded by the decoder 964, i.e., decompressed using a suitable technique, and then 
displayed at client computer 240 by renderer 965. ") (col 9, lines 1-10) for playback (i.e. when the video 
data/frames are displayed or audio data played (i.e. multimedia data) at the client computer) (Abstract; Fig.3; 
Fig.4B; Fig.SB; Fig.9; Fig.lOA) as 3 media Stream (i.e. "The video/audio streams and annotation streams are 
stored in stream server(s) for subsequent retrieval by client computer(s) in a coordinated manner, so that the client 
computer(s) is able to synchronously display the video frames and displayable event(s) in a video window and event . 

window(s), respectively. ") (col 3, lines 1-20), a method for encoding a data segment (i.e. the 

annotation frame) (Abstract; Fig.3; Fig.4B; Fig.SB; Fig.9; Fig.lOA; col 3, lines 1-25) and time restriction 
information (i.e. "Another visual control aid, zoom bar 716, enables designer 219 to select the respective 
portions of the complete time tracks 750, 760, 770, 780, and 790, and defined by start time indicator 712 and end 
time indicator 718, which is currently displayed by author tool 700. ") (col 7, lines 10-67) for Said Segment 

with said media file (col 2, lines 25-67) such that when said encoded media file is decoded 

for playback (Abstract; Fig.3; Fig.4B; Fig.SB; Fig.9; Fig.lOA) 3S said media Stream (col 3, lines 1-20), 
Said data segment (i.e. the annotation frame) (Abstract; Fig.3; Fig.4B; Fig.SB; Fig.9; Fig.lOA; col 3, lines 1- 
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25) is reproduced at a specified point in said media stream only during a time period (i.e. 

"For example, URL addresses (event locators) pointing to HTML pages enable client computer 240 to subsequently 
retrieve textual and/or graphical elements to be displayed at predetermined time as defined by the time markers of 

the flipper stream. ") (col 7, lines 10-67) predetermined by said time restriction information (col i, 
lines 10-67), comprising the steps of: (a) receiving a media file (col 2, lines 25-67); (b) receiving 

a Segment Of data (i.e. the annotation frame) (Abstract; Fig.3; Fig.4B; Fig.8B; Fig.9; Fig.lOA); (c) 

receiving the specifics of said time period when said data segment (col 3, lines 1-25) is to 
be reproduced, including one of the following: i) a start time (i.e. the start time) (col 1, lines 10- 

67), ii) a Stop time (i.e. the end time) (col 7, lines 10-67), Hi) both 3 Start and a Stop time (col 7, lines 
10-67). 

Pumaveja does not explicitly teach embedding data segment as part of said 

encoded media data. 

Tao teaches embedding data segment as part of said encoded media data (i.e. 

-The present invention pertains to a method and system thereof for invisibly encoding and embedding information 
within digital data representing, for example, audio data, image data, and video data. " ... "In one embodime3nt, 
the digital data are in a compressed format (e.g. JPEG MPEG for image data, MP3 orAC3for audio data, as well 
as other data compress ion formats.) ") (Abstract; col 2, lines 28-45; col 3, lines 1-10). 

It would have been obvious to a person of ordinary skill in the art at the time of 
Applicant's invention to modify the teachings of Purnaveja with the teachings of Tao to 
include embedding data segment as part of said encoded media data with the 
motivation to embed multiple speech streams into a video, each speech stream in a 
different language, so that the video can be distributed to a wide range of users who 
can then each view the video in the language of their choosing (Tao, col 1, lines 30-35) and 
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to store a large amount of information while being robust against noise (Tao, col 4, lines 33- 



As to claim 1 1 , Purnaveja teaches that the media file is an audio file (i.e. "Annotated 

multimedia streams can include a compressed video stream for display in a video window, an accompanying 
compressed audio stream and annotations. " ... "Sources for generating video streams include a video camera 312, 
a video cassette recorder (VCR) (not shown) or a previously digitized video file 314, e.g., a Video for Windows 
(.avi) fde. ") (Examiner asserts that Purnaveja explicitly states that the system and the associated method are 
applicable to video, audio and image data. A video file example is given, but it is also applicable to audio media 

fiie(s)) (col 2, lines 35-67; col 5, lines 54-64) and said data segment (col 3, lines 1-25) is a non-audio 

data Segment (i.e. "The first type of annotation streams are data annotation streams in which the displayable 
event data are embedded within the annotation streams. Examples of data annotation streams include ticker tape 
data embedded within the annotation stream. ") (col 6, lines 25-67). 

As to claim 12, Purnaveja teaches a decoder (i.e. the decoder 964) (col 9, lines l-io) for 

decoding an encoded media file (i.e. "a previously digitized video file 3 1 4, e.g., a Video for Windows 
(avi) file. ") (col 2, lines 25-67) for playback as a media Stream (i.e. "The video/audio streams and 

annotation streams are stored in stream server(s) for subsequent retrieval by client computer(s) in a coordinated 
manner, so that the client computer(s) is able to synchronously display the video frames and displayable event(s) in 
a video window and event window(s), respectively. ") (col 3, lines 1-20), a method for decoding data (i.e. 
"Annotation interpreter 963 parses annotation frames in real time in the form of messages from stream server 
220. ") (col 9, lines 1-25) and time restriction information (i.e. "Another visual control aid, zoom bar 716, 

enables designer 219 to select the respective portions of the complete time tracks 750, 760, 770, 780, and 790, and 
defined by start time indicator 712 and end time indicator 718, which is currently displayed by author tool 700. ") 



36). 
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(col 7, lines 10-67), comprising the steps of: (1) receiving a block (i.e. frame) of said encoded 

media file (col 2, lines 25-67); (2) decoding a data Segment (i.e. ••Annotation interpreter 963 parses 
annotation frames in real time in the form of messages from stream server 220. ") (col 3, lines 1-25; col 9, lines 1- 

25); (3) decoding said time restriction information (col i, lines io-67) embedded (col 6, lines 55- 

67; col 7, lines 14-28) within Said data Segment (i.e. the annotation frame) (Abstract; Fig.3; Fig.4B; Fig.8B; 

Fig.9; Fig.iOA; col 3, lines 1-25), wherein said time restriction information (col i, lines 10-67) 

Specifies a time period (i.e. "For example, URL addresses (event locators) pointing to HTML pages enable 
client computer 240 to subsequently retrieve textual and/or graphical elements to be displayed at predetermined 
time as defined by the time markers of the flipper stream. ") (col 7, lines 10-67) When Said data Segment (col 

3, lines 1-25) is to be reproduced with said decoded media stream (i.e. "The video/audio streams 

and annotation streams are stored in stream server(s) for subsequent retrieval by client computer(s) in a 
coordinated manner, so that the client computer(s) is able to synchronously display the video frames and 
display able event (s) in a video window and event window '(s), respectively. ") (col 3, lines 1-20); and (4) 
reproducing Said data Segment (i.e. the annotation frame) (Abstract; Fig.3; Fig.4B; Fig.8B; Fig.9; 

Fig.iOA; col 3, lines 1-25) at a specified point in said decoded media stream for playback only 
during said specified time period (col 7, lines io-67). 

Purnaveja does not explicitly teach embedding data segment as part of said 
encoded media data. 

Tao teaches embedding data segment as part of said encoded media data (i.e. 

"The present invention pertains to a method and system thereof for invisibly encoding and embedding information 
within digital data representing, for example, audio data, image data, and video data. " ... "In one embodime3nt, 
the digital data are in a compressed format (e.g. JPEG MPEG for image data, MP3 or AC3 for audio data, as well 
as other data compression formats.) ") (Abstract; col 2, lines 28-45; col 3, lines 1-10). 
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It would have been obvious to a person of ordinary skill in the art at the time of 
Applicant's invention to modify the teachings of Purnaveja with the teachings of Tao to 
include embedding data segment as part of said encoded media data with the 
motivation to embed multiple speech streams into a video, each speech stream in a 
different language, so that the video can be distributed to a wide range of users who 
can then each view the video in the language of their choosing (Tao, col i, lines 30-35) and 
to store a large amount of information while being robust against noise (Tao, col 4, lines 33- 

36). 



As to claim 1 3, Purnaveja teaches that the reproducing of said data segment (col 
3, lines 1-25) includes subsequent processing for enabling the data to be displayed to a 

User (i.e. The second type of annotation streams are locator annotation streams in which the displayable data is 
either too cumbersome and/or is continually evolving to be embedded as static data within the annotation stream. 
Instead, event locator(s) pointing to the location of the displayable data are stored in the annotation streams instead 
of the displayable data. Examples include URL addresses pointing to HTML pages. " ... "In this example, URL 
addresses, for synchronizing HTML page flips with video stream are provided to web browser 950 thereby 
permitting client computer 240 to subsequently retrieve and display various textual and graphical elements 
changing at predetermined points corresponding to the timeline of the video stream. Note that HTML pages can be 
retrieved from one or more web server (s) 230. ") (Examiner asserts that the client needs user input (or client has to 
determine if the user wants to reproduce the HTML pages in the web browser using the URL/ internet address 
provided in the annotation frames) to retrieve the HTML pages from the web server(s)) (col 6, lines 50-67; col 9, 
lines 45-67). 



Application/Control Number: 09/884,504 Pa 9 e 1 3 

Art Unit: 2175 

As to claim 22, Purnaveja teaches a system for encoding data (i.e. the annotation data) 

and time restriction information (i.e. "Another visual control aid, zoom bar 716, enables designer 219 to 
select the respective portions of the complete time tracks 750, 760, 770, 780, and 790, and defined by start time 
indicator 712 and end time indicator 718, which is currently displayed by author tool 700. ") (col 7, lines 10-67) 
With 3 media file (i.e. "a previously digitized video file 314, e.g., a Video for Windows (.avi)file. ") (col 2, lines 

25-67) and for the later decoding of said encoded media file such that when said encoded 

media file is decoded for playback (i.e. when the video data/frames are displayed or audio data played 
(i.e. multimedia data) at the client computer) (Abstract; Fig.3; Fig.4B; Fig.SB; Fig.9; Fig.lOA) 3S 3 media 
Stream (i.e. "The video/audio streams and annotation streams are stored in stream server(s)for subsequent 
retrieval by client computer(s) in a coordinated manner, so that the client computers) is able to synchronously 
display the video frames and displayable event(s) in a video window and event window(s), respectively. ") (col 3, 

lines i-20), said data is reproduced in said media stream only during a time period 

predetermined (i.e. "For example, URL addresses (event locators) pointing to HTML pages enable client 
computer 240 to subsequently retrieve textual and/or graphical elements to be displayed at predetermined. time as 
defined by the time markers of the flipper stream. ") (col 7, lines 10-67) by Said time restriction 

information (col 7, lines io-67), comprising: an encoder (col 6, lines 25-35) operative to receive 

Said media file (col 2, lines 25-67), at least One data file (i.e. "When author module 318 has completed 
building an annotation stream, e.g., the flipper stream, the annotation stream is given a file name and loaded into a 
convenient server, e.g., stream server 220, for subsequent retrieval by client computer 240. ") (col 7, lines 10-67) 
Comprising at least One data segment (i.e. the annotation frame) (Abstract; Fig.3; Fig.4B; Fig.SB; 

Fig.9; Fig.i0A; col 3, lines 1-25), and time restriction information (col 7, lines 10-67) for each said 

data Segment (i.e. the annotation frame) (Abstract; Fig.3; Fig.4B; Fig.SB; Fig.9; Fig.lOA; col 3, lines 1-25); 

and a decoder (i.e. the decoder 964) (col 9, lines i-io) operative to decode said encoded media 
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file (col 2, lines 25-67) into said decoded media stream (col 3, lines 1-20), said data segment (i.e. 

the annotation frame) (Abstract; Fig.3; Fig.4B; Fig.8B; Fig.9; Fig.lOA; col 3, lines 1-25) and Said time 

restriction information (col i, lines io-67), and operative to playback (i.e. when the video 

data/frames are displayed or audio data played (i.e. multimedia data) at the client computer) (Abstract; Fig.3; 
Fig.4B; Fig.8B; Fig.9; Fig.lOA) Said data segment (i.e. the annotation frame) (Abstract; Fig.3; Fig.4B; 

Fi g .8B; Fig.9; Fig.lOA; col 3, lines 1-25) as part of said decoded media stream only during said 

time period (i.e. "For example, URL addresses (event locators) pointing to HTML pages enable client computer 
240 to subsequently retrieve textual and/or graphical elements to be displayed at predetermined time as defined by 

the time markers of the flipper stream. ") (col 7, lines 10-67) specified by said corresponding time 
restriction information (col 7, lines 10-67). 

Purnaveja does not explicitly teach embedding data segment as part of said 
encoded media data. 

Tao teaches embedding data segment as part of said encoded media data (i.e. 

"The present invention pertains to a method and system thereof for invisibly encoding and embedding information 
within digital data representing, for example, audio data, image data, and video data. " ... "In one embodime3nt, 
the digital data are in a compressed format (e.g. JPEG MPEG for image data, MP3 or AC3 for audio data, as well 
as other data compression formats.) ") (Abstract; col 2, lines 28-45; col 3, lines 1-10). 

It would have been obvious to a person of ordinary skill in the art at the time of 
Applicant's invention to modify the teachings of Purnaveja with the teachings of Tao to 
include embedding data segment as part of said encoded media data with the 
motivation to embed multiple speech streams into a video, each speech stream in a 
different language, so that the video can be distributed to a wide range of users who 
can then each view the video in the language of their choosing (Tao, col i, lines 30-35) and 
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to store a large amount of information while being robust against noise (Tao, col 4, lines 33- 

36). 

As to claim 23, Purnaveja teaches a method for encoding (i.e. "a previously digitized 

video file 314, e.g., a Video for Windows (.avi) file. ") (col 2, lines 25-67) and thereafter decoding (i.e. 
"Next, the encoded video/audio streams are decoded by the decoder 964, i.e., decompressed using a suitable 
technique, and then displayed at client computer 240 by renderer 965. ") (col 9, lines 1-10) a media file (i.e. "a 
previously digitized video file 314, e.g., a Video for Windows (.avi) file. ") (col 2, lines 25-67) SUCh that a data 
Segment (i.e. the annotation frame) (Abstract; Fig.3; Fig.4B; Fig.8B; Fig.9; Fig.lOA; col 3, lines 1-25) and 

said media file after encoding is reproduced during decoding at a specified point during 

a predetermined time period (i.e. "For example, URL addresses (event locators) pointing to HTML pages 
enable client computer 240 to subsequently retrieve textual and/or graphical elements to be displayed at 
predetermined time as defined by the time markers of the flipper stream. ") (col 7, lines 10-67) for playback (i.e. 
when the video data/frames are displayed or audio data played (i.e. multimedia data) at the client computer) 
(Abstract; Fig.3; Fig.4B; Fig.8B; Fig.9; Fig.lOA) as a media Stream (i.e. "The video/audio streams and 
annotation streams are stored in stream server(s) for subsequent retrieval by client computers) in a coordinated 
manner, so that the client computer(s) is able to synchronously display the video frames and displayable event(s) in 
a video window and event window(s), respectively. ") (col 3, lines 1-20) for a USer, Comprising the Steps 
Of: (1 ) receiving a media file (i.e. "a previously digitized video file 314, e.g., a Video for Windows (.avi) 
file. ") (col 2, lines 25-67); (2) receiving at least One data file (i.e. "When author module 318 has 
completed building an annotation stream, e.g., the flipper stream, the annotation stream is given a file name and 
loaded into a convenient server, e.g., stream server 220, for subsequent retrieval by client computer 240. ") (col 7, 
lines 10-67) comprising at least One data segment (i.e. the annotation frame) (Abstract; Fig.3; Fig.4B; 
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Fi g .8B; Fig.9; Fig.lOA; col 3, lines 1-25); (3) receiving time restriction information for each said 

data Segment (i.e. the annotation frame) (Abstract; Fig.3; Fig.4B; Fig.8B; Fig.9; Fig.lOA; col 3, lines 1-25) 
that Specifies Said time period (i.e. "For example, URL addresses (event locators) pointing to HTML 
pages enable client computer 240 to subsequently retrieve textual and/or graphical elements to be displayed at 
predetermined time as defined by the time markers of the flipper stream. ") (col 7, lines 10-67) When Said data 
Segment (Abstract; Fig.3; Fig.4B; Fig.SB; Fig.9; Fig.lOA; col 3, lines 1-25) is to be reproduced 3S part 
Of Said decoded media Stream (i.e. "The video/audio streams and annotation streams are stored in stream 
server(s) for subsequent retrieval by client computers) in a coordinated manner, so that the client computers) is 
able to synchronously display the video frames and displayable event(s) in a video window and event window(s), 

respectively. ") (col 3, lines 1-20); and (4) receiving encoded media file (col 2, lines 25-67); and said 

data segment (Abstract; Fig.3; Fig.4B; Fig.8B; Fig.9; Fig.lOA; col 3, lines 1-25) and Said 
Corresponding time restriction information (i.e. "Another visual control aid, zoom bar 716, enables 
designer 219 to select the respective portions of the complete time tracks 750, 760, 770, 780, and 790, and defined 
by start time indicator 712 and end time indicator 718, which is currently displayed by author tool 700. ") (col 7, 

lines io-67) such that said data segment (col 3, lines 1-25) is reproduced by a decoder (i.e. the 
decoder 964) (col 9, lines l-io) during said specified time period (col i, lines io-67) (5) receiving a 

blOCk (i.e. the audio/video frame) (Abstract) Of Said encoded media file (col 8, lines 59-67); (6) 
decoding (i.e. interpreting the encoded annotated frame by annotation interpreter 963) (col 9, lines 10-25) Said 

data segment (col 3, lines 1-25); (7) decoding said time restriction information (col 7, lines 10- 

67) embedded (i.e. "The first type of annotation streams are data annotation streams in which the displayable 
event data are embedded within the annotation streams. Examples of data annotation streams include ticker tape 
data embedded within the annotation stream. ") (col 6, lines 49-67) Within Said block for Said data 
Segment (i.e. interpreting the encoded annotated frame by annotation interpreter 963) (col 3, lines 1-25; col 9, 
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lines 10-25), wherein said time restriction information (col i, lines io-67) specifies a time period 
(col 7, lines 10-67) when said data segment (col 3, lines 1-25) is to be reproduced as part of 

Said decoded media Stream (i.e. "The video/audio streams and annotation streams are stored in stream 
server(s) for subsequent retrieval by client computers) in a coordinated manner, so that the client computer® is 
able to synchronously display the video frames and displayable event(s) in a video window and event window(s), 
respectively. ") (col 3, lines 1-20); and (8) reproducing Said data Segment (i.e. intetpreting the encoded 
annotated frame by annotation interpreter 963) (col 3, lines 1-25; col 9, lines 10-25) at a specified point in 
Said decoded media Stream for playback (i.e. when the video data/frames are displayed or audio data 
played (i.e. multimedia data) at the client computer) (Abstract; Fig.3; Fig.4B; Fig.SB; Fig.9; Fig.lOA) Only 

during said specified time period (col i, lines io-67) . 

Purnaveja does not explicitly teach embedding data segment as part of said 

encoded media data. 

Tao teaches embedding data segment as part of said encoded media data (i.e. 

"The present invention pertains to a method and system thereof for invisibly encoding and embedding information 
within digital data representing, for example, audio data, image data, and video data. " ... "In one embodiment, 
the digital data are in a compressed format (e.g. JPEG MPEG for image data, MPS or ACS for audio data, as well 
as other data compression formats.) ") (Abstract; col 2, lines 28-45; col 3, lines 1-10). 

It would have been obvious to a person of ordinary skill in the art at the time of 
Applicant's invention to modify the teachings of Purnaveja with the teachings of Tao to 
include embedding data segment as part of said encoded media data with the 
motivation to embed multiple speech streams into a video, each speech stream in a 
different language, so that the video can be distributed to a wide range of users who 
can then each view the video in the language of their choosing (Tao, col 1, lines 30-35) and 
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to store a large amount of information while being robust against noise (Tao, col 4, lines 33- 

36). 

3. Claims 14-21 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Purnaveja et al. (U.S. Patent No. 6,006,241 and Purnaveja hereinafter) in view of Tao 
(U.S. Patent No. 6,243,481) and Sonoda et al. (U.S. Patent No. 6,557,171 and Sonoda 
hereinafter). 

The teachings of Purnaveja and Tao have been discussed previously in the 
above rejections. 

As to claim 14, Purnaveja teaches a decoder (i.e. the decoder 964) (col 9, lines l-io) for 

decoding an encoded media file (i.e. "a previously digitized video file 314, e.g., a Video for Windows 
(.avijfile. ") (col 2, lines 25-67) for playback 3S 3 media Stream (i.e. "The video/audio streams and 
annotation streams are stored in stream server® for subsequent retrieval by client computer® in a coordinated 
manner, so that the client computer® is able to synchronously display the video frames and displayable event(s) in 
a video window and event window, respectively.") (col 3, lines 1-20), 3 method for decoding (i.e. 
interpreting the encoded annotated frame by annotation interpreter 963) (col 9, lines 10-25) an embedded (i.e. 
"The first type of annotation streams are data annotation streams in which the displayable event data are embedded 
within the annotation streams. Examples of data annotation streams include ticker tape data embedded within the 
annotation stream. ") (col 6, lines 49-67) data segment (i.e. the annotation frame) (Abstract; Fig.3; Fig.4B; 
Fig.8B; Fig.9; Fig.lOA; col 3, lines 1-25) and time restriction information (i.e. "Another visual control aid, 
zoom bar 716, enables designer 219 to select the respective portions of the complete time tracks 750, 760, 770, 780, 
and 790, and defined by start time indicator 712 and end time indicator 718, which is currently displayed by author 
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tool 100. 1 (col 7, toes .0-67) from said encoded media file (,, ~a previously di S i,ned video f,ic 3,4, 
e . e ., a video for f «w* 1 M 2, u- 25*> such that a data segment ,,e. the annoution 
frame) (Abstract; Kg .3; F ig .4B; *** Fig .9; F ig ,0A; co, 3, lines ,-2 5 , wherein said data segment is to 
be reproduced at a specified point in said media stream (c„i 3, lines 1-20) only during a time 

period predetermined (i., -For cample, URL addresses (even, locators) points ,0 HTML pases enable 
client computer 240 to subseauen.ly relieve ,e*,ua, and,or g raph,cal elements to Be displayed a, predefined 
^^fr*^^**/"^^™""* 7 -*" ' M7>by time reS,riCti0n 

information (col 7, hne, .0-67), comprising the steps of: (a) reading a block (,., v,deo/a„d,„ 
frame) (col s, lines 59-67) of said encoded media file (col 2, imes 25-67,: (b) determining whether 

said data Segment (Le. annotation frame) (col 3, lines 1-25) exists (i.e. "Ticker annotation Jran.es are 
provided by ,he strean, server 220 synchronously ,„h tlte vUieo/audlo frames so tka, ,he annotations, i.e., 
displayable ticker data can be synchronized for display a, client computer 240 a, predetermined points 
correspond^ to the timeltne of. he video stream. ") (The Examiner asserts that the client retrieve, and displays the 
anno.at.on data according to the timeline. The Cen. firs, has to firs, determine if there is any annotation frame,. If 
toe is no annotation frames, the client does no, ha»e to display annotation data, it j»s. plays the video/an* stream 
data) (col 9, line, 40-67); (O if said data segment (i.e. annotation frame) (col 3, lines 1-25) does not 
exist, then proceeding to Step (P) (i.e. -Ticker anno,a,,on frames are provided by, he s^eam server 220 
synchronously «k the videoMudiofra.es so tha, the annotations, i.e.. displayable ticker data can be synchronised 
for display at client computer 240 at predetermined points correspond^ to the timeline of, he video stream. ■> (The 
Examine, asserts tha, the Cent retneves and display, the annotation data acc„rdin g to the .imeline. The client firs, 
has ,o ft* determine if mere i, any anno«ation frame, If mere is no annotation frame,, me clien, doe, no, have ,o 
di,p,ay anno«,ion data, i, jus, play, ,he video/audio „ream da,a, (col 9, lines 40-67); (d) determining 

whether a start time tag (i... M time indi.a,or, (on, 7, lines 9-20) exists in said data segment ,co. 

3, line, 1-25) that specifies When said time period is to begin (i.e. 'Another visual control aid. zoom 
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bar 716, enables designer 219 to select the respective portions of the complete time tracks 750, 760, 770, 780, and 
790, and defined by start time indicator 712 and end time indicator 718, which is currently displayed by author tool 
700. " ... "Ticker annotation frames are provided by the stream server 220 synchronously with the video/audio 
frames so that the annotations, i.e., displayable ticker data can be synchronized for display at client computer 240 at 
predetermined points corresponding to the timeline of the video stream. ") (The Examiner asserts that the client 
receives and displays the annotation data synchronously with the audio/video data. The client first determines if 
there is an annotation frame to be reproduced and if there is any timeline associated with the annotation frame. 
Therefore if there is a timeline (the timeline consists of the difference between start time and end time) associated 
with the annotation frame, the annotation frame data is reproduced at the client according to the timeline 
synchronously with the audio/video data, otherwise (i.e. there is no start time/end time) at the same time with the 
video/audio data frames. If there is no annotation frame, there is no annotation data to reproduce, the client just 
plays the video/audio stream data) (col 7, lines 10-67; col 9, lines 45-67); (e) if said Start time tag (i.e. start 
time indicator) (col 7, lines 9-20) does not exist, then proceeding tO Step Q) (i e. "Another visual 
control aid, zoom bar 716, enables designer 219 to select the respective portions of the complete time tracks 750, 
760, 770, 780, and 790, and defined by start time indicator 712 and end time indicator 718, which is currently 
displayed by author tool 700. "... "Ticker annotation frames are provided by the stream server 220 synchronously 
with the video/audio frames so that the annotations, i.e., displayable ticker data can be synchronized for display at 
client computer 240 at predetermined points corresponding to the timeline of the video stream. ") (The Examiner 
asserts that the client receives and displays the annotation data synchronously with the audio/video data. The client 
first determines if there is an annotation frame to be reproduced and if there is any timeline associated with the 
annotation frame. Therefore if there is a timeline (the timeline consists of the difference between start time and end 
time) associated with the annotation frame, the annotation frame data is reproduced at the client according to the 
timeline synchronously with the audio/video data, otherwise (i.e. there is no start time indicator/end time indicator) 
at the same time with the video/audio data frames. If there is no annotation frame, there is no annotation data to 
reproduce, the client just plays the video/audio stream data) (col 7, lines 10-67; col 9, lines 45-67); (f) 

determining the start time specified by said start time tag (i.e. "Another visual control aid, zoom 
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bar 716, enables designer 219 to select the respective portions of the complete time tracks 750, 760, 770, 780, and 
790, and defined by start time indicator 712 and end time indicator 718, which is currently displayed by author tool 
700. "... "Ticker annotation frames are provided by the stream server 220 synchronously with the video/audio 
frames so that the annotations, i.e., displayable ticker data can be synchronized for display at client computer 240 at 
predetermined points corresponding to the timeline of the video stream. ") (The Examiner asserts that the client 
receives and displays the annotation data synchronously with the audio/video data. The client first determines if 
there is an annotation frame to be reproduced and if there is any timeline associated with the annotation frame. 
Therefore if there is a timeline (the timeline consists of the difference between start time and end time) associated 
with the annotation frame, the annotation frame data is reproduced at the client according to the timeline 
synchronously with the audio/video data, otherwise (i.e. there is no start time indicator/end time indicator) at the 
same time with the video/audio data frames. If there is no annotation frame, there is no annotation data to reproduce, 
the client just plays the video/audio stream data) (col 7, lines 10-67; col 9, lines 45-67); Q) determining 

whether a stop time tag (i.e. end time indicator) (col 7, lines 9-20) exists in said data segment (col 

3, lines 1-25) that Specifies When Said time period is tO end (i.e. "Another visual control aid, zoom 
bar 716, enables designer 219 to select the respective portions of the complete time tracks 750, 760, 770, 780, and 
790, and defined by start time indicator 712 and end time indicator 718, which is currently displayed by author tool 
700. "... "Ticker annotation frames are provided by the stream server 220 synchronously with the video/audio 
frames so that the annotations, i.e., displayable ticker data can be synchronized for display at client computer 240 at 
predetermined points corresponding to the timeline of the video stream. ") (The Examiner asserts that the client 
receives and displays the annotation data synchronously with the audio/video data. The client first determines if 
there is an annotation frame to be reproduced and if there is any timeline associated with the annotation frame. 
Therefore if there is a timeline (the timeline consists of the difference between start time and end time) associated 
with the annotation frame, the annotation frame data is reproduced at the client according to the timeline 
synchronously with the audio/video data, otherwise (i.e. there is no start time indicator/end time indicator) at the 
same time with the video/audio data frames. If there is no annotation frame, there is no annotation data to reproduce, 
the client just plays the video/audio stream data) (col 7, lines 10-67; col 9, lines 45-67); (k) if said Stop time 
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tag does not exist, then proceeding tO Step (O) (i.e. "Another visual control aid, zoom bar 716, 
enables designer 219 to select the respective portions of the complete time tracks 750, 760, 770, 780, and 790, and 
defined by start time indicator 712 and end time indicator 718, which is currently displayed by author tool 700. "... 
"Ticker annotation frames are provided by the stream server 220 synchronously with the video/audio frames so that 
the annotations, i.e., displayable ticker data can be synchronized for display at client computer 240 at 
predetermined points corresponding to the timeline of the video stream. ") (The Examiner asserts that the client 
receives and displays the annotation data synchronously with the audio/video data. The client first determines if 
there is an annotation frame to be reproduced and if there is any timeline associated with the annotation frame. 
Therefore if there is a timeline (the timeline consists of the difference between start time and end time) associated 
with the annotation frame, the annotation frame data is reproduced at the client according to the timeline 
synchronously with the audio/video data, otherwise (i.e. there is no start time indicator/end time indicator) at the 
same time with the video/audio data frames. If there is no annotation frame, there is no annotation data to reproduce, 
the client just plays the video/audio stream data) (col 7, lines 10-67; col 9, lines 45-67); (I) determining the 
Stop time Specified in Said Stop time tag (i.e. "Another visual control aid, zoom bar 716, enables 
designer 219 to select the respective portions of the complete time tracks 750, 760, 770, 780, and 790, and defined 
by start time indicator 712 and end time indicator 718, which is currently displayed by author tool 700. "... "Ticker 
annotation frames are provided by the stream server 220 synchronously with the video/audio frames so that the 
annotations, i.e., displayable ticker data can be synchronized for display at client computer 240 at predetermined 
points corresponding to the timeline of the video stream. ") (The Examiner asserts that the client receives and 
displays the annotation data synchronously with the audio/video data. The client first determines if there is an 
annotation frame to be reproduced and if there is any timeline associated with the annotation frame. Therefore if 
there is a timeline (the timeline consists of the difference between start time and end time) associated with the 
annotation frame, the annotation frame data is reproduced at the client according to the timeline synchronously with 
the audio/video data, otherwise (i.e. there is no start time indicator/end time indicator) at the same time with the 
video/audio data frames. If there is no annotation frame, there is no annotation data to reproduce, the client just 
plays the video/audio stream data) (col 7, lines 10-67; col 9, lines 45-67); (o) reproducing Said data 
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Segment (i.e. "Ticker annotation frames are provided by the stream server 220 synchronously with the 
video/audio frames so that the annotations, i.e., displayable ticker data can be synchronized for display at client 
computer 240 at predetermined points corresponding to the timeline of the video stream. ") (The Examiner asserts 
that the client receives and displays the annotation data synchronously with the audio/video data. The client first 
determines if there is an annotation frame to be reproduced and if there is any timeline associated with the 
annotation frame. Therefore if there is a timeline (the timeline consists of the difference between start time and end 
time) associated with the annotation frame, the annotation frame data is reproduced at the client according to the 
timeline synchronously with the audio/video data, otherwise (i.e. there is no start time indicator/end time indicator) 
at the same time with the video/audio data frames. If there is no annotation frame, there is no annotation data to 
reproduce, the client just plays the video/audio stream data) (col 7, lines 10-67; col 9, lines 45-67); and (p) 
Continuing tO Step (a) (i.e. The Examiner asserts that the client moves on to the next frame in the stream) (col 
7, lines 10-67; col 9, lines 45-67). 

Purnaveja does not explicitly teach embedding data segment as part of said 

encoded media data. 

Tao teaches embedding data segment as part of said encoded media data (i.e. 

"The present invention pertains to a method and system thereof for invisibly encoding and embedding information 
within digital data representing, for example, audio data, image data, and video data. "... "In one embodime3nt, 
the digital data are in a compressed format (e.g. JPEG MPEG for image data, MP3 orAC3for audio data, as well 
as other data compression formats.) ") (Abstract; col 2, lines 28-45; col 3, lines 1-10). 

It would have been obvious to a person of ordinary skill in the art at the time of 
Applicant's invention to modify the teachings of Purnaveja with the teachings of Tao to 
include embedding data segment as part of said encoded media data with the 
motivation to embed multiple speech streams into a video, each speech stream in a 
different language, so that the video can be distributed to a wide range of users who 
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can then each view the video in the language of their choosing (Tao, col l, lines 30-35) and 
to store a large amount of information while being robust against.noise (Tao, col 4, lines 33- 

36). 

Purnaveja does not explicitly teach (g) getting the current time; (h) comparing 
said start time to the current time; (i) if the current time is before said start time then 
proceeding to step (p); (m) comparing said stop time tag to the current time; (n) if the 
current time is after said stop time then proceeding to step (p). 

Sonoda teaches (g) getting the current time (col 22, line 67; col 23, lines 1-26); (h) 
comparing said start time (col 22, line 67; col 23, lines 1-26) to the current time (i.e. "The reception 

control unit 130 asks the timing managing unit 132 the current time, and compares the obtained " start Jime" with 

the current time'') (col 22, line 67; col 23, lines 1-26); (i) if the current time is before said start time 

then proceeding tO Step (p) (i.e. "The reception control unit 130 asks the timing managing unit 132 the 
current time, and compares the obtained "start Jime" with the current time. When the current time is in the period 
between the "start Jime" and the "start time* duration, " the reception control unit 130 immediately refers to the 
system tables corresponding to the latest reselected program stored in the system table storage unit 127, and 
instructs TS decoder unit 122 to extract the presentation data of the program. ") (Abstract; col 22, line 67; col 23, 
lines 1-26); (m) comparing Said Stop time (i.e. start_time + duration) (col 22, line 67; col 23, lines 1-26) 
tag to the Current time (i.e. "The reception control unit 130 asks the timing managing unit 132 the current 
time, and compares the obtained "start Jime " with the current time. When the current time is in the period between 
the "start Jime" and the "start time+ duration, " the reception control unit 130 immediately refers to the system 
tables corresponding to the latest reselected program stored in the system table storage unit 127, and instructs TS 
decoder unit 122 to extract the presentation data of the program. ") (Abstract; col 22, line 67; col 23, lines 1-26); 
(n) if the Current time (i.e. "The reception control unit 130 asks the timing managing unit 132 the current 
time, and compares the obtained "start Jime" with the current time. When the current time is in the period between 
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the ' 'start Jime " and the "start time+ duration, " the reception control unit 130 immediately refers to the system 
tables corresponding to the latest reselected program stored in the system table storage unit 127, and instructs TS 
decoder unit 122 to extract the presentation data of the program. ") (Abstract; col 22, line 67; col 23, lines 1-26) IS 
after Said Stop time (i.e. "start time+ duration") (Abstract; col 22, line 67; col 23, lines 1-26) then 
proceeding to Step (p) (Abstract; col 22, line 67; col 23, lines 1-26). 

It would have been obvious to a person of ordinary skill in the art at the time of 
Applicant's invention to modify the teachings of Purnaveja and Tao with the teachings of 
Sonoda to include (g) getting the current time; (h) comparing said start time to the 
current time; (i) if the current time is before said start time then proceeding to step (p); 
(m) comparing said stop time tag to the current time; (n) if the current time is after said 
stop time then proceeding to step (p) with the motivation to broadcast the promotional 
program (i.e. annotation data) in the same broadcast periods as the object programs (i.e. 

audio/video data in Purnaveja) (Sonoda,; col 23, lines 1-3) and to facilitate the manipulation Of the 

viewer for pre-selecting programs (Sonoda, col 8, lines 63-64) . 

As to claim 15, Purnaveja teaches that the media file is an audio file (i.e. "Annotated 

multimedia streams can include a compressed video stream for display in a video window, an accompanying 
compressed audio stream and annotations. "... "Sources for generating video streams include a video camera 312, 
a video cassette recorder (VCR) (not shown) or a previously digitized video file 314, e.g., a Video for Windows 
(avi) file. ") (Examiner asserts that Purnaveja explicitly states that the system and the associated method are 
applicable to video, audio and image data. A video file example is given, but it is also applicable to audio media 

fiie(s)) (col 2, lines 35-67; col 5, lines 54-64) and said data segment (col 3, lines 1-25) is a non-audio 

data Segment (i.e. "The first type of annotation streams are data annotation streams in which the displayable 
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event data are embedded within the annotation streams. Examples of data annotation streams include ticker tape 
data embedded within the annotation stream. ") (col 6, lines 25-67). 

Purnaveja does not explicitly teach embedding data segment as part of said 

encoded media data. 

Tao teaches embedding data segment as part of said encoded media data (i.e. 

"The present invention pertains to a method and system thereof for invisibly encoding and embedding information 
within digital data representing, for example, audio data, image data, and video data. " ... "In one embodime3nt, 
the digital data are in a compressed format (e.g. JPEG MPEG for image data, MPS orAC3for audio data, as well 
as other data compression formats.) ") (Abstract; col 2, lines 28-45; col 3, lines 1-10). 

It would have been obvious to a person of ordinary skill in the art at the time of 
Applicant's invention to modify the teachings of Purnaveja with the teachings of Tao to 
include embedding data segment as part of said encoded media data with the 
motivation to embed multiple speech streams into a video, each speech stream in a 
different language, so that the video can be distributed to a wide range of users who 
can then each view the video in the language of their choosing (Tao, col 1, lines 30-35) and 
to store a large amount of information while being robust against noise (Tao, col 4, lines 33- 

36). 

As to claim 16, Purnaveja teaches the step of proceeding to step (o) if no current 

time is available (i.e. "Ticker annotation frames are provided by the stream server 220 synchronously with the 
video/audio frames so that the annotations, i.e., displayable ticker data can be synchronized for display at client 
computer 240 at predetermined points corresponding to the timeline of the video stream. ") (The Examiner asserts 
that the client receives and displays the annotation data synchronously with the audio/video data. The client first 
determines if there is an annotation frame to be reproduced and if there is any timeline associated with the 
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annotation frame. Therefore if there is a timeline (the timeline consists of the difference between start time and end 
time) associated with the annotation frame, the annotation frame data is reproduced at the client according to the 
timeline synchronously with the audio/video data, otherwise (i.e. there is no start time indicator/end time indicator or 
there is no current time provided by the client system there is no way of verifying the time restriction and hence the 
annotation data is reproduced synchronously with the media data) at the same time with the video/audio data frames. 
If there is no annotation frame, there is no annotation data to reproduce, the client just plays the video/audio stream 
data) (col 7, lines 10-67; col 9, lines 45-67). 

As to claim 17, Purnaveja teaches the step of proceeding to step (p) if no current 

time iS available (i.e. "Ticker annotation frames are provided by the stream server 220 synchronously with the 
video/audio frames so that the annotations, i.e., displayable ticker data can be synchronized for display at client 
computer 240 at predetermined points corresponding to the timeline of the video stream. ") (The Examiner asserts 
that the client receives and displays the annotation data synchronously with the audio/video data. The client first 
determines if there is an annotation frame to be reproduced and if there is any timeline associated with the 
annotation frame. Therefore if there is a timeline (the timeline consists of the difference between start time and end 
time) associated with the annotation frame, the annotation frame data is reproduced at the client according to the 
timeline synchronously with the audio/video data, otherwise (i.e. there is no start time indicator/end time indicator or 
there is no current time provided by the client system there is no way of verifying the time restriction and hence the 

annotation data is reproduced synchronously with the media data) at the same time with the video/audio data frames. 

If there is no annotation frame, there is no annotation data to reproduce, the client just plays the video/audio stream 

data and the client moves on reading the new audio/video frame transmitted by the stream server) (col 7, lines 10- 

67; col 9, lines 45-67). 

As to claim 18, Purnaveja teaches that the step (o) further includes the step of 

determining Whether Said segment (i.e. the annotation frame) (Abstract; Fig.3; Fig.4B; Fig.8B; Fig.9; 
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Fig.lOA) iS reproduced based On input from a USer (i.e. "Flipper annotation frames are provided by 
stream server220 synchronously with the video/audio frames to the client module 960 so that the annotations, i.e., 
displayable events can be synchronized for display at client computer 24 (step 1 060). In this example, URL 
addresses, for synchronizing HTML page flips with video stream are provided to web browser 950 thereby 
permitting client computer 240 to subsequently retrieve and display various textual and graphical elements 
changing at predetermined points corresponding to the timeline of the video stream. Note that HTML pages can be 
retrieved from one or more web server(s) 230. ") (Examiner asserts that the client needs user input (or client has to 
determine if the user wants to reproduce the HTML pages in the web browser using the URL/ internet address 
provided in the annotation frames) to retrieve the HTML pages from the web server(s)) (col 7, lines 10-67; col 9, 

lines 45-67). 

Purnaveja does not explicitly teach embedding data segment as part of said 

encoded media data. 

Tao teaches embedding data segment as part of said encoded media data (i.e. 

"The present invention pertains to a method and system thereof for invisibly encoding and embedding information 
within digital data representing, for example, audio data, image data, and video data. "... "In one embodiment, 
the digital data are in a compressed format (e.g. JPEG MPEG for image data, MP3 orAC3for audio data, as well 
as other data compression formats.) ") (Abstract; col 2, lines 28-45; col 3, lines 1-10). 

It would have been obvious to a person of ordinary skill in the art at the time of 
Applicant's invention to modify the teachings of Purnaveja with the teachings of Tao to 
include embedding data segment as part of said encoded media data with the 
motivation to embed multiple speech streams into a video, each speech stream in a 
different language, so that the video can be distributed to a wide range of users who 
can then each view the video in the language of their choosing (Tao, col 1, lines 30-35) and 
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to store a large amount of information while being robust against noise (Tao, col 4, lines 33- 

36). 

As to claim 19, Purnaveja teaches the reproducing of said data segment (i.e. 

interpreting the encoded annotated frame by annotation interpreter 963) (Abstract; Fig.3; Fig.4B; Fig.8B; Fig.9; 

Fig.iOA; col 3, lines 1-25; col 9, lines 10-25) in step (o) includes subsequent processing for 

enabling the data to be displayed tO a User (i.e. The second type of annotation streams are locator 
annotation streams in which the displayable data is either too cumbersome and/or is continually evolving to be 
embedded as static data within the annotation stream. Instead, event locator(s) pointing to the location of the 
displayable data are stored in the annotation streams instead of the displayable data. Examples include URL 
addresses pointing to HTML pages. " ... "In this example, URL addresses, for synchronizing HTML page flips with 
video stream are provided to web browser 950 thereby permitting client computer 240 to subsequently retrieve and 
display various textual and graphical elements changing at predetermined points corresponding to the timeline of 
the video stream. Note that HTML pages can be retrieved from one or more web server (s) 230. ") (Examiner asserts 
that the client needs user input (or client has to determine if the user wants to reproduce the HTML pages in the web 
browser using the URL/ internet address provided in the annotation frames) to retrieve the HTML pages from the 
web server(s)) (col 6, lines 50-67; col 9, lines 45-67). 

Purnaveja does not explicitly teach embedding data segment as part of said 

encoded media data. 

Tao teaches embedding data segment as part of said encoded media data (i.e. 

"The present invention pertains to a method and system thereof for invisibly encoding and embedding information 
within digital data representing, for example, audio data, image data, and video data. " ... "In one embodime3nt, 
the digital data are in a compressed format (e.g. JPEG MPEG for image data, MP3 orAC3for audio data, as well 
as other data compression formats.)") (Abstract; col 2, lines 28-45; col 3, lines 1-10). 
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It would have been obvious to a person of ordinary skill in the art at the time of 
Applicant's invention to modify the teachings of Purnaveja with the teachings of Tao to 
include embedding data segment as part of said encoded media data with the 
motivation to embed multiple speech streams into a video, each speech stream in a 
different language, so that the video can be distributed to a wide range of users who 
can then each view the video in the language of their choosing (Tao, col l, lines 30-35) and 
to store a large amount of information while being robust against noise (Tao, col 4, lines 33- 

36). 

As to claim 20, Purnaveja teaches that the data segment (i.e. the annotation frame) (col 

9, lines 10-25) is an Internet address (i.e. "Examples include URL addresses pointing to HTML pages. ") (col 
6, lines 49-67). 

Purnaveja does not explicitly teach embedding data segment as part of said 
encoded media data. 

Tao teaches embedding data segment as part of said encoded media data (i.e. 

"The present invention pertains to a method and system thereof for invisibly encoding and embedding information 
within digital data representing, for example, audio data, image data, and video data. " ... "In one embodime3nt, 
the digital data are in a compressed format (e.g. JPEG MPEG for image data, MP3 orAC3for audio data, as well 
as other data compression formats.) ") (Abstract; col 2, lines 28-45; col 3, lines 1-10). 

It would have been obvious to a person of ordinary skill in the art at the time of 
Applicant's invention to modify the teachings of Purnaveja with the teachings of Tao to 
include embedding data segment as part of said encoded media data with the 
motivation to embed multiple speech streams into a video, each speech stream in a 
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different language, so that the video can be distributed to a wide range of users who 
can then each view the video in the language of their choosing (Tao, col l, lines 30-35) and 
to store a large amount of information while being robust against noise (Tao, col 4, lines 33- 

36). 

As to claim 21 , Purnaveja teaches that the processing includes linking to an 
Internet document (i.e. html pages) (col 6, lines 49-67) determined by said Internet address (i.e. 

"Examples include URL addresses pointing to HTML pages. ") (col 6, lines 49-67). 



Points of Contact 

4. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Apu M. Mofiz whose telephone number is (703) 605- 
4240. The examiner can normally be reached on Monday - Thursday 8:00 A.M. to 4:30 
P.M. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Dov Popovici can be reached at (703) 305-3830. The fax numbers for the 
group is (703) 872-9306. 
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